Syntactically coherent code segmentation
Techniques are described herein for segmenting source code into syntactically coherent sequences of tokens that satisfy constraints inherent in sequence-to-sequence networks. In various implementations, source code may be processed to generate one or more graphs representing the source code. One or more of the graphs may then be traversed to identify one or more sequences of tokens within the source code that satisfy an input constraint of a sequence-to-sequence network. The source code may be segmented into the identified one or more sequences of tokens. The one or more sequences of tokens may then be processed using the sequence-to-sequence network.
1. A method implemented by one or more processors and comprising:
processing source code to generate one or more graphs representing the source code;
traversing one or more of the graphs to identify one or more sequences of tokens within the source code that satisfy an input constraint of a trained machine learning model comprising a transformer network with an attention mechanism, wherein the input constraint comprises a limit on how many tokens can be processed during a single iteration of the transformer network;
segmenting the source code into the identified one or more sequences of tokens; and
processing the one or more sequences of tokens using the transformer network.
2. The method of claim 1 , wherein the one or more graphs include an abstract syntax tree (AST) representing the source code.
3. The method of claim 1 , wherein the one or more graphs include at least one of a data flow graph (DFG) or a control flow graph (CFG) representing the source code.
4. The method of claim 1 , wherein the traversing one or more of the graphs to identify the one or more sequences of tokens includes:
determining a count of tokens represented by a node of one or more of the graphs; and
comparing the count of tokens to the input constraint of the transformer network.
5. The method of claim 4 , wherein determining the count includes:
identifying a portion of the source code represented by the node; and
consulting an interval tree based on the identified portion of the source code to determine the count.
6. The method of claim 4 , wherein the node is a first node, the count is a first count, and the method further comprises:
in response to a determination that the first count of tokens satisfies the input constraint of the transformer network, determining a second count of tokens represented by a second node of one or more of the graphs, and comparing the second count of tokens to the input constraint of the transformer network.
7. The method of claim 6 , wherein the second node comprises a sibling node of the first node.
8. The method of claim 4 , wherein the node is a first node, the count is a first count, and the method further comprises:
in response to a determination that the first count of tokens fails to satisfy the input constraint of the transformer network, determining a second count of tokens represented by a child node of the first node, and comparing the second count of tokens to the input constraint of the transformer network.
9. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:
process source code to generate one or more graphs representing the source code;
traverse one or more of the graphs to identify one or more sequences of tokens within the source code that satisfy an input constraint of a trained machine learning model comprising a transformer network with an attention mechanism, wherein the input constraint comprises a limit on how many tokens can be processed during a single iteration of the transformer network;
segment the source code into the identified one or more sequences of tokens; and
process the one or more sequences of tokens using the transformer network.
10. The system of claim 9 , wherein the one or more graphs include an abstract syntax tree (AST) representing the source code.
11. The system of claim 9 , wherein the one or more graphs include at least one of a data flow graph (DFG) or a control flow graph (CFG) representing the source code.
12. The system of claim 9 , wherein the instructions to traverse one or more of the graphs to identify the one or more sequences of tokens include instructions to:
determine a count of tokens represented by a node of one or more of the graphs; and
compare the count of tokens to the input constraint of the transformer network.
13. The system of claim 12 , wherein the instructions to determine the count include instructions to:
identify a portion of the source code represented by the node; and
consult an interval tree based on the identified portion of the source code to determine the count.
14. The system of claim 12 , wherein the node is a first node, the count is a first count, and the system further comprises instructions to:
in response to a determination that the first count of tokens satisfies the input constraint of the transformer network, determine a second count of tokens represented by a second node of one or more of the graphs, and compare the second count of tokens to the input constraint of the transformer network.
15. The system of claim 14 , wherein the second node comprises a sibling node of the first node.
16. The system of claim 12 , wherein the node is a first node, the count is a first count, and the system further comprises instructions to:
in response to a determination that the first count of tokens fails to satisfy the input constraint of the transformer network, determine a second count of tokens represented by a child node of the first node, and compare the second count of tokens to the input constraint of the transformer network.
17. At least one non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to:
process source code to generate one or more graphs representing the source code;
traverse one or more of the graphs to identify one or more sequences of tokens within the source code that satisfy an input constraint of a trained machine learning model comprising a transformer network with an attention mechanism, wherein the input constraint comprises a limit on how many tokens can be processed during a single iteration of the transformer network;
segment the source code into the identified one or more sequences of tokens; and
process the one or more sequences of tokens using the transformer network.
18. The at least one non-transitory computer-readable medium of claim 9 , wherein the one or more graphs include at least one of a data flow graph (DFG) or a control flow graph (CFG) representing the source code.