Training and/or using machine learning model(s) for automatic generation of test case(s) for source code
Training and/or utilization of machine learning model(s) (e.g., neural network model(s)) in automatically generating test case(s) for source code. Techniques disclosed herein can be utilized in generating test case(s) for unit test testing (or other white-box testing) and/or for functional testing (or other black-box testing).
1 . A method implemented by one or more processors, the method comprising:
identifying a source code unit that is programmed in a programming language; processing the source code unit, using a code-to-embedding machine learning model, to generate one or more code unit embeddings in a latent space;
processing the one or more code unit embeddings, using a code embedding-to-test embedding machine learning model, to generate one or more predicted unit test embeddings in the latent space;
using the one or more predicted unit test embeddings to identify a unit test for the source code unit; and
evaluating the source code unit using at least the unit test, wherein evaluating the source code unit using at least the unit test comprises:
executing the source code unit, using test inputs of the unit test, to generate source code unit output, and
evaluating the source code unit based on the source code unit output.
2 . The method of claim 1 , wherein using the one or more predicted unit test embeddings to identify the unit test for the source code unit comprises:
processing the one or more predicted unit test embeddings, using an embedding-to-code machine learning model, to generate the unit test for the source code unit.
3 . The method of claim 2 , wherein processing the one or more predicted unit test embeddings, using the embedding-to-code machine learning model, to generate the unit test for the source code unit comprises:
generating a sequence of outputs based on processing the one or more predicted unit test embeddings using the embedding-to-code machine learning model, wherein the outputs of the sequence each comprise a corresponding probability distribution; and
generating the unit test based on highest probabilities in the corresponding probability distributions of the sequence of outputs.
4 . The method of claim 1 , wherein using the one or more predicted unit test embeddings to identify the unit test for the source code unit comprises:
determining that the one or more predicted unit test embeddings satisfy a distance threshold relative to one or more pre-generated unit test embeddings for the unit test, wherein the one or more pre-generated unit test embeddings are generated by processing the unit test using the code-to-embedding machine learning model; and
responsive to determining that the one or more predicted unit test embeddings satisfy the distance threshold relative to the one or more pre-generated unit test embeddings for the unit test:
identifying the unit test.
5 . The method of claim 1 , further comprising:
determining, based on the evaluating, to deploy the source code unit in production.
6 . The method of claim 1 , further comprising:
causing rendering, in a development application, of one or more metrics that are based on the evaluating.
7 . The method of claim 1 , wherein the source code unit is in a second programming language and is an automated translation of a corresponding source code unit in a first programming language that is different from the second programming language, and further comprising:
determining, based on the evaluating, whether to render the source code unit in a development application as a suggested translation of the corresponding source code unit.
8 . The method of claim 1 , wherein the source code unit is one of multiple candidate automated translations of the corresponding source code unit and wherein determining, based on the evaluating, whether to render the source code unit in the development application as the suggested translation comprises:
determining, based on the evaluating, whether to render the source code unit, or an alternate one of the automated translations, as a suggested translation of the corresponding source code unit.
9 . The method of claim 1 , wherein the code embedding-to-test embedding machine learning model is a transformer neural network model or an encoder-decoder neural network model.
10 . A method implemented by one or more processors, the method comprising:
identifying base source code that is programmed in a base programming language;
identifying target source code that is in a target programming language and that is generated based on automatically translating the base source code;
processing the base source code, using a code-to-embedding machine learning model, to generate a base source code embedding in a latent space;
processing the target source code, using the code-to-embedding machine learning model, to generate a target source code embedding in the latent space;
processing, using a policy network, the base source code embedding and the target source code embedding, to generate a predicted test input;
executing the base source code using the predicted test input to generate a base output;
executing the target source code using the predicted test input to generate a target output; and
determining, based on comparing the base output and the target output, whether one or more branches of the target source code are a valid translation of the base source code.
11 . The method of claim 10 , wherein determining, based on comparing the base output and the target output, whether one or more branches of the target source code are a valid translation of the base source code, comprises:
determining, based on the base output and the target output not being equal, that at least a given branch, involved in executing the target source code, is not a valid translation.
12 . The method of claim 11 , further comprising:
responsive to determining that at least the given branch, involved in executing the target source code, is not a valid translation:
causing the given branch to be rendered, in a development application, with a particular demarcation.
13 . The method of claim 10 , wherein the policy network is trained, through reinforcement learning, to reward predictions of test inputs that result in differing outputs when used in testing a base source code, target source code pair.
14 . The method of claim 10 , further comprising:
automatically translating the base source code to generate the target source code, wherein identifying the target source code is responsive to automatically translating the base source code to generate the target source code.
15 . A system, comprising:
memory storing instructions;
one or more processors operable to execute the instructions to:
identify a source code unit that is programmed in a programming language;
process the source code unit, using a code-to-embedding machine learning model, to generate one or more code unit embeddings in a latent space;
process the one or more code unit embeddings, using a code embedding-to-test embedding machine learning model, to generate one or more predicted unit test embeddings in the latent space;
use the one or more predicted unit test embeddings to identify a unit test for the source code unit; and
evaluate the source code unit using at least the unit test, wherein in evaluating the source code unit using at least the unit test one or more of the processors are to:
execute the source code unit, using test inputs of the unit test, to generate source code unit output, and
evaluate the source code unit based on the source code unit output.
16 . The system of claim 15 , wherein in using the one or more predicted unit test embeddings to identify the unit test for the source code unit one or more of the processors are to:
process the one or more predicted unit test embeddings, using an embedding-to-code machine learning model, to generate the unit test for the source code unit.
17 . The system of claim 16 , wherein in processing the one or more predicted unit test embeddings, using the embedding-to-code machine learning model, to generate the unit test for the source code unit one or more of the processors are to:
generate a sequence of outputs based on processing the one or more predicted unit test embeddings using the embedding-to-code machine learning model, wherein the outputs of the sequence each comprise a corresponding probability distribution; and
generate the unit test based on highest probabilities in the corresponding probability distributions of the sequence of outputs.
18 . The system of claim 15 , wherein one or more of the processors are further operable to execute the instructions to:
determine, based on the evaluating, to deploy the source code unit in production.
19 . The system of claim 15 , wherein one or more of the processors are further operable to execute the instructions to:
cause rendering, in a development application, of one or more metrics that are based on the evaluating.