Automating test-driven development with transformers
A test-driven development system utilizes a neural transformer model with attention to generate method bodies for a focal method given its associated test cases, and optionally a method signature and a docstring of the focal method. The candidate method bodies are validated for syntactic correctness, tested using the given test cases, and tested with a donor class in a target system. Those candidate method bodies passing the validation and testing are then ranked based on a PLUM score that analyzes the candidate method bodies against various quality and performance metrics.
1. A system comprising:
one or more processors; and
a memory that stores one or more programs that are configured to be executed by the one or more processors, the one or more programs include instructions to perform actions that:
obtain a plurality of test cases for testing a focal method, wherein source code of a method body of the focal method has not been developed;
generate, from a deep learning model given the plurality of test cases, a method signature and/or docstring of the focal method, at least one candidate method body;
test the at least one candidate method body for syntactic correctness;
upon successful validation of the at least one candidate method body for syntactic correctness, test the at least one candidate method body with the plurality of test cases;
upon the at least one candidate method body successfully passing the plurality of test cases, compute one or more code quality metrics for the at least one candidate method body and rank the at least one candidate method body based on the one or more code quality metrics; and output the ranked at least one candidate method body.
2. The system of claim 1 , wherein the one or more programs include further instructions to perform actions that:
upon the at least one candidate method body failing the syntactic correctness validation, eliminate the at least one candidate method body.
3. The system of claim 1 , wherein the one or more programs include further instructions to perform actions that:
upon the at least one candidate method body failing the plurality of test cases, eliminate the at least one candidate method body.
4. The system of claim 1 , wherein the one or more programs include further instructions to perform actions that:
prior to outputting the at least one candidate method body, compile the at least one candidate method body with a donor class; and
upon the at least one candidate method body failing to compile, eliminate the at least one candidate method body.
5. The system of claim 1 , wherein the deep learning model is a neural transformer model with attention having at least one encoder block coupled to at least one decoder block.
6. The system of claim 5 , wherein the neural transformer model with attention is pre-trained on natural language text and source code and fine-tuned on a plurality of test cases for a plurality of target methods.
7. A computer-implemented method, comprising:
accessing a plurality of test cases for a method in a software development environment, wherein source code for a method body of the method has not been implemented;
utilizing a deep learning model to generate a plurality of candidate method bodies for the method given the plurality of test cases, a method signature and/or docstring;
validating each of the plurality of candidate method bodies for syntactic correctness and compliance with the plurality of test cases;
upon successful validation and compliance of select ones of the plurality of candidate method bodies, computing one or more code quality metrics for each of the select ones of the plurality of candidate method bodies and rank the select ones of the plurality of candidate method bodies based on the one or more code quality metrics; and
outputting the select ones of the plurality of candidate method bodies in the software development environment.
8. The computer-implemented method of claim 7 , wherein validating each of the plurality of candidate method bodies, further comprises:
compiling each of the plurality of candidate method bodies using a donor class; and
eliminating select ones of the plurality of candidate method bodies that fail to compile.
9. The computer-implemented method of claim 7 , wherein the one or more quality metrics include cyclomatic complexity metric, code size metric, maintainability index metric, and/or code coupling and cohesion metric.
10. The computer-implemented method of claim 7 , wherein the deep learning model is a neural transformer model with attention.