System and method for code smell detection using transformer-based code representations with self-supervision by predicting reserved words
A device, method, and non-transitory computer readable medium that for analyzing computer source code to detect code smells is disclosed. The method includes inputting, via processing circuitry, the source code and creating, via the processing circuitry, pseudo labels by a proxy task based on a vector of tokens for the source code. In addition, the method includes training, via the processing circuitry, a transformer model on the pseudo labels, as a pre-trained model that outputs a prediction of a value of tokens in the vector of tokens, and applying, via the processing circuitry, the pre-trained model to a plurality of fine-tuning models for respective downstream tasks, where each fine-tuning model is created by training the pre-trained model. The method also includes outputting, via the processing circuitry, from each fine-tuning model, an indication of whether a code smell has been detected in the source code.
1 . A method of analyzing computer source code to detect code smells, comprising:
inputting, via processing circuitry, the source code;
creating, via the processing circuitry, pseudo labels by a proxy task based on a vector of tokens for the source code;
training, via the processing circuitry, a transformer model on the pseudo labels, as a pre-trained model that outputs a prediction of a value of tokens in the vector of tokens;
applying, via the processing circuitry, the pre-trained model to a plurality of fine-tuning models for respective downstream tasks, wherein each fine-tuning model is created by training the pre-trained model; and
outputting, via the processing circuitry, from each fine-tuning model, an indication of whether a code smell has been detected in the source code.
2 . The method of claim 1 , further comprising:
inputting, via the processing circuitry, the source code;
parsing, via the processing circuitry, the source code to extract classes and extract textual features of the classes;
applying, via the processing circuitry, a set of preprocessing techniques, including tokenization, filtering, truncation, and vectorization, to convert the textual features into a proper format; and
inputting, via the processing circuitry, the formatted textual features for training the transformer model.
3 . The method of claim 2 , wherein the step of creating pseudo labels includes:
given results from the preprocessing techniques of the formatted textual features that consists of m instances where a single instance i is associated with x value which is a feature vector of length n tokens, where each instance i represents a whole class or part of a class, a list R of reserved words of a target programming language, and a unique token MASK ID :
getting, via the processing circuitry, index positions of reserved words in a feature vector of n tokens;
masking, via the processing circuitry, reserved words randomly;
masking, via the processing circuitry, random tokens; and
constructing, via the processing circuitry, a new data set with the pseudolabels.
4 . The method of claim 3 , wherein the training includes:
inputting, via the processing circuitry, the randomly masked reserved words; and
training, via the processing circuitry, the Transformer model on the source code in which the masked reserved words have been applied.
5 . The method of claim 1 , wherein the respective downstream tasks include:
class-level code smells: God Class and Data Class, and method-level code smells: Feature Envy and Long Method.
6 . The method of claim 1 , wherein the inputting includes
inputting, via the processing circuitry, a sequence of code for a class, wherein the sequence of code is a sequence of tokens including out of vocabulary tokens and mask tokens.
7 . The method of claim 6 , wherein a representation of each token in the sequence of code is a summation of a token embedding and a positional embedding.
8 . The method of claim 7 , wherein in the token embedding, each token is mapped to a dense vector of a fixed embedding dimension.
9 . The method of claim 7 , wherein the positional embedding is a representation of a position of each token within the sequence of code.
10 . A system for source code smell detection, comprising: processing circuitry configured to
input a source code,
create pseudo labels by a proxy task based on a vector of tokens for the source code,
train a transformer model on the pseudo labels, as a pre-trained model that outputs a prediction of a value of tokens in the vector of tokens,
apply the pre-trained model to a plurality of fine-tuning models for respective downstream tasks, wherein each fine-tuning model is created by training the pre-trained model, and
output from each fine-tuning model, an indication of whether a code smell has been detected in the source code.
11 . The system of claim 10 , the processing circuitry further configured to:
input the source code,
parse the source code to extract classes and extract textual features of the classes,
apply a set of preprocessing techniques, including tokenization, filtering, truncation, and vectorization, to convert the textual features into a proper format, and
input the formatted textual features for training the transformer model.
12 . The system of claim 11 , wherein the create pseudo labels by the processing circuitry includes:
given results from the preprocessing techniques, of the formatted textual features that consists of m instances where a single instance i is associated with x value which is a feature vector of length n tokens, where each instance i represents a whole class or part of a class, a list R of reserved words of a target programming language, and a unique token MASK ID : the processing circuitry further configured to:
get index positions of reserved words in a feature vector of n tokens,
mask reserved words randomly,
mask random tokens, and
construct a new data set with the pseudo labels.
13 . The system of claim 12 , wherein the train by the processing circuitry is further configured to:
input the randomly masked reserved words, and
train the Transformer model on the source code in which the masked reserved words have been applied.
14 . The system of claim 10 , wherein the respective downstream tasks, by the processing circuitry, include:
class-level code smells: God Class and Data Class, and method-level code smells: Feature Envy and Long Method.
15 . The system of claim 10 , wherein the input by the processing circuitry includes:
input a sequence of code for a class, wherein the sequence is a sequence of tokens including out of vocabulary tokens and mask tokens.
16 . The system of claim 15 , wherein a representation of each token in the sequence of tokens is a summation of a token embedding and a positional embedding.
17 . The system of claim 16 , wherein in the token embedding, each token is mapped to a dense vector of a fixed embedding dimension.
18 . The system of claim 17 , wherein the positional embedding is a representation of a position of each token within the sequence of tokens.
19 . The system of claim 10 , wherein the respective downstream tasks that the pre-trained model is applied are each configured on separate processing circuitry, wherein each fine-tuning model for a respective downstream task is created by training the pre-trained model.
20 . A non-transitory computer-readable storage medium including computer executable instructions, wherein the instructions, when executed by processing circuitry, cause the processing circuitry to perform a method of analyzing computer source code to detect code smells, the method comprising:
inputting, via the processing circuitry, the source code;
creating, via the processing circuitry, pseudo labels by a proxy task based on a vector of tokens for the source code;
training, via the processing circuitry, a transformer model on the pseudo labels, as a pre-trained model that outputs a prediction of a value of tokens in the vector of tokens;
applying, via the processing circuitry, the pre-trained model to a plurality of fine-tuning models for respective downstream tasks, wherein each fine-tuning model is created by training the pre-trained model; and
outputting, via the processing circuitry, from each fine-tuning model, an indication of whether a code smell has been detected in the source code.