Similarity scoring of programs
View Patent ↗A technique is provided for determining a similarity score between a source program and a test program. The technique includes generating an abstract syntax tree for both programs and converting the subtrees of each syntax tree into hash vectors. Vectors in the abstract syntax tree for the source program are dimension reduced and made accessible in a spatial data structure such as an R-tree. The structure is searched using the hash vectors from the test program abstract syntax tree and candidate matching vectors are found. A search is conducted among the candidate matching vectors for pairs that have a degree of similarity above a given threshold and these are the reported results.
1. A method for generating a similarity score between a source program and a test program, the method comprising:
generating a source abstract syntax tree (sAST) for a source program and a test abstract syntax tree (tAST) for a test program;
generating hash vectors for each abstract syntax tree (AST) by hashing subtrees of the sAST and tAST;
reducing the dimensionality hash vectors of the sAST;
forming a spatial data structure based on the reduced dimensionality hash vectors;
querying the spatial data structure with the hash vectors of the tAST to find candidate matching vectors;
searching among the candidate matching vectors for pairs of vectors that exceed a similarity threshold; and
reporting the pairs that exceed the similarity threshold.
2. The method of claim 1 , wherein the spatial data structure is an R-tree.
3. The method of claim 1 , wherein the spatial data structure is a ball tree.
4. The method of claim 1 , wherein searching among the candidate matching vectors for pairs of vectors that exceed a similarity threshold includes:
generating pairs of matching vectors from the candidate matching vectors; and
merging similar pairs of matching vectors to form a merged set that contains pairs that exceed the similarity threshold.
5. The method of claim 4 , wherein merging similar pairs of matching vectors includes:
comparing a left similarity factor and right similarity factor against a threshold; and
comparing a left betweenness factor and a right betweenness factor against the threshold.
6. The method of claim 5 , wherein the left similarity factor is based on a left common value and a left difference value.
7. The method of claim 5 , wherein the right similarity factor is based on a right common value and a right difference value.
8. The method of claim 5 , wherein the left betweenness factor is based on a left common value and a size of a set of left line numbers.
9. The method of claim 5 , wherein the right betweenness factor is based on a right common value and a size of a set of right line numbers.
10. The method of claim 1 , wherein searching among the candidate matching vectors for pairs of vectors that exceed a similarity threshold includes:
comparing the source AST tree with the test AST by traversing the source AST from a leaf of the tree and computing a comparison score at each level of the tree; and
reporting a pair of vectors as being similar if the comparison score exceeds a threshold when traversal reaches the root of the source AST.
11. A non-transitory computer-readable storage medium comprising instructions that, when executed in a computing device, enable a method for generating a similarity score between a source program and a test program, by performing the steps of:
generating a source abstract syntax tree (sAST) for a source program and a test abstract syntax tree (tAST) for a test program;
generating hash vectors for each abstract syntax tree (AST) by hashing subtrees of the sAST and tAST;
reducing the dimensionality hash vectors of the sAST;
forming a spatial data structure based on the reduced dimensionality hash vectors;
querying the spatial data structure with the hash vectors of the tAST to find candidate matching vectors;
searching among the candidate matching vectors for pairs of vectors that exceed a similarity threshold; and
reporting the pairs that exceed the similarity threshold.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the spatial data structure is an R-tree.
13. The non-transitory computer-readable storage medium of claim 11 , wherein the spatial data structure is a ball tree.
14. The non-transitory computer-readable storage medium of claim 11 , wherein searching among the candidate matching vectors for pairs of vectors that exceed a similarity threshold includes instructions that when executed perform the steps of:
generating pairs of matching vectors from the candidate matching vectors; and
merging similar pairs of matching vectors to form a merged set that contains pairs that exceed the similarity threshold.
15. The non-transitory computer-readable storage medium of claim 14 , wherein merging similar pairs of matching vectors includes instructions that when executed perform the steps of:
comparing a left similarity factor and right similarity factor against a threshold; and
comparing a left betweenness factor and a right betweenness factor against the threshold.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the left similarity factor is based on a left common value and a left difference value.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the right similarity factor is based on a right common value and a right difference value.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the left betweenness factor is based on a left common value and a size of a set of left line numbers.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the right betweenness factor is based on a right common value and a size of a set of right line numbers.
20. The non-transitory computer-readable storage medium of claim 11 , wherein searching among the candidate matching vectors for pairs of vectors that exceed a similarity threshold includes instructions that when executed perform the steps of:
comparing the source AST tree with the test AST by traversing the source AST from a leaf of the tree and computing a comparison score at each level of the tree; and
reporting a pair of vectors as being similar if the comparison score exceeds a threshold when traversal reaches the root of the source AST.