Systems and methods for detecting code duplication in codebases

1. A method for detecting exact code snippet duplicates, comprising:

representing, by a code duplication detection computer program, each of a plurality of code snippets in a codebase as abstract syntax trees;

applying, by the code duplication detection computer program, a de-noising filter to the plurality of code snippets or the abstract syntax trees;

featurizing, by the code duplication detection computer program, the abstract syntax trees into corpus feature vectors by converting the abstract syntax tree into vector representations;

generating, by the code duplication detection computer program, dense feature vectors from the corpus feature vectors using a dimension reduction technique;

clustering, by the code duplication detection computer program, the dense feature vectors into dendrograms, each dendrogram having a different value for a cluster distance metric;

applying, by the code duplication detection computer program, cross-correlation thresholding to identify an optimal value for the cluster distance metric;

applying, by the code duplication detection computer program, iterative density-based clustering to the dendrogram for the optimal value for the cluster distance metric;

identifying, by the code duplication detection computer program, exact duplicate code snippet matches by apply density-based clustering to the dense feature vectors; and

tagging, by the code duplication detection computer program, the exact duplicate code snippets.

2. The method of claim 1 , further comprising:

applying, by the code duplication detection computer program, Natural Language Processing (NLP) to generate features for the abstract syntax trees.

3. The method of claim 1 , wherein the de-noising filter filters code snippets or abstract syntax trees that are not actively used.

4. The method of claim 1 , wherein the de-noising filter filters code snippets or abstract syntax trees that are irrelevant.

5. The method of claim 1 , wherein the de-noising filter is based on a trained neural network.

6. The method of claim 1 , wherein the corpus feature vectors comprise a list of featurized abstract syntax trees from a code corpus.

7. The method of claim 1 , wherein the dimension reduction technique comprises truncated Singular Value Decomposition.

8. A method for detecting near code snippet duplicates, comprising:

representing, by a code duplication detection computer program, each of a plurality of code snippets in a codebase as abstract syntax trees;

applying, by the code duplication detection computer program, a de-noising filter to the plurality of code snippets or the abstract syntax trees;

featurizing, by the code duplication detection computer program, the abstract syntax trees into corpus feature vectors by converting the abstract syntax trees into vector representations;

generating, by the code duplication detection computer program, dense feature vectors from the corpus feature vectors using a dimension reduction technique;

clustering, by the code duplication detection computer program, the dense feature vectors into dendrograms, each dendrogram having a different value for a cluster distance metric;

applying, by the code duplication detection computer program, cross-correlation thresholding to identify an optimal value for the cluster distance metric;

applying, by the code duplication detection computer program, iterative density-based clustering to the dendrogram for the optimal value for the cluster distance metric;

tracking, by the code duplication detection computer program, data points in the dendrogram that have merged into a large cluster but were also present in small unique clusters, wherein the data points belonging to the unique small cluster to identify code snippets that are near duplicates of each other; and

tagging, by the code duplication detection computer program, the near duplicate code snippets.

9. The method of claim 8 , further comprising:

applying, by the code duplication detection computer program, Natural Language Processing (NLP) to generate features for the abstract syntax trees.

10. The method of claim 8 , wherein the de-noising filter filters code snippets or abstract syntax trees that are not actively used.

11. The method of claim 8 , wherein the de-noising filter is based on a trained neural network.

12. The method of claim 8 , wherein the corpus feature vectors comprise a list of featurized abstract syntax trees from a code corpus.

13. The method of claim 8 , wherein the dimension reduction technique comprises truncated Singular Value Decomposition.

14. A method for detecting exact code snippet duplicates, comprising:

loading, by a code duplication detection computer program, a near duplicate centroid, an exact duplicate centroid, a vectorizer, and a dimension reduction model;

producing, by the code duplication detection computer program, dense vectors from incremental functions;

representing, by a code duplication detection computer program, each of a plurality of incremental functions as abstract syntax trees;

featurizing, by the code duplication detection computer program, the abstract syntax trees into incremental function feature vectors by converting the abstract syntax trees into vector representations;

generating, by the code duplication detection computer program, dense feature vectors from the incremental function feature vectors using the dimension reduction model;

computing, by the code duplication detection computer program, a cosine similarity between the generated dense vectors and the near duplicate centroid and the exact duplicate centroid;

ranking, by the code duplication detection computer program, the near duplicate centroid and the exact duplicate centroid in descending order based on the similarity;

thresholding, by the code duplication detection computer program, the ranked near duplicate centroid and the exact duplicate centroid;

selecting, by the code duplication detection computer program, a top most ranked centroid; and

identifying, by the code duplication detection computer program, the incremental function as a duplicate of data points in a cluster of the top-most centroid.

15. The method of claim 14 , wherein the dimension reduction model comprises truncated Singular Value Decomposition.

16. The method of claim 14 , further comprising:

applying, by the code duplication detection computer program, Natural Language Processing (NLP) to generate features for the abstract syntax trees.

17. The method of claim 14 , further comprising:

applying, by the code duplication detection computer program, a de-noising filter to the plurality of the abstract syntax trees.

18. The method of claim 17 , wherein the de-noising filter filters abstract syntax trees that are irrelevant.