Training syntax-aware language models with AST path prediction
In an embodiment, a computer stores and operates a logic encoder that is an artificial neural network that infers a fixed-size encoded logic from textual or tokenized source logic. Without machine learning, a special parser generates a parse tree that represents the source logic and a fixed-size correctly encoded tree that represents the parse tree. For finetuning the logic encoder, an encoded tree generator is an artificial neural network that accepts the fixed-size encoded logic as input and responsively infers a fixed-size incorrectly encoded tree that represents the parse tree. The neural weights of the logic encoder (and optionally of the encoded tree generator) are adjusted based on backpropagation of error (i.e. loss) as a numerically measured difference between the fixed-size incorrectly encoded tree and the fixed-size correctly encoded tree.
1 . A method comprising:
increasing accuracy of a logic encoder that is a bidirectional encoder by finetuning the logic encoder, including repeating in a predetermined count of repetitions:
a) inferring, by the logic encoder, a fixed-size encoded logic from a source logic;
b) generating a parse tree that represents the source logic;
c) generating a fixed-size correctly encoded tree that represents the parse tree that represents the source logic;
d) inferring, based on the fixed-size encoded logic, by an encoded tree generator, a fixed-size incorrectly encoded tree that represents the parse tree that represents the source logic; and
e) neural backpropagating, in the logic encoder, a difference between the fixed-size incorrectly encoded tree and the fixed-size correctly encoded tree; and
performing after said increasing the accuracy of the logic encoder:
deploying, into a production environment, the logic encoder without the encoded tree generator;
generating, by the logic encoder, a second fixed-size encoded logic for a second source logic in an integrated development environment (IDE); and
performing, by the IDE based on the second fixed-size encoded logic, source logic completion;
wherein the method is performed by one or more computers.
2 . The method of claim 1 wherein:
said inferring the fixed-size incorrectly encoded tree is performed by the encoded tree generator;
the method further comprises adjusting the encoded tree generator based on said difference between the fixed-size incorrectly encoded tree and the fixed-size correctly encoded tree.
3 . The method of claim 1 further comprising sizing, based on a count of distinct tree paths in parse trees in a training corpus, the fixed-size correctly encoded tree and the fixed-size incorrectly encoded tree.
4 . The method of claim 1 wherein said inferring the fixed-size incorrectly encoded tree comprises inferring a plurality of distinct tree paths in the parse tree that represents the source logic.
5 . The method of claim 4 wherein said inferring the plurality of distinct tree paths comprises inferring a respective count of each tree path in the plurality of distinct tree paths.
6 . The method of claim 1 wherein said generating the fixed-size correctly encoded tree is based on at least one selected from a group consisting of: a predefined minimum path length, a predefined maximum path length, and non-terminal nodes of the parse tree that represents the source logic but not terminal nodes.
7 . The method of claim 1 further comprising based on the second fixed-size encoded logic, performing at least one selected from a group consisting of: source logic translation, source logic documentation generation, and programing reference material recommendation.
8 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:
increasing accuracy of a logic encoder that is a bidirectional encoder by finetuning the logic encoder, including repeating in a predetermined count of repetitions:
a) inferring, by the logic encoder, a fixed-size encoded logic from a source logic;
b) generating a parse tree that represents the source logic;
c) generating a fixed-size correctly encoded tree that represents the parse tree that represents the source logic;
d) inferring, based on the fixed-size encoded logic, by an encoded tree generator, a fixed-size incorrectly encoded tree that represents the parse tree that represents the source logic; and
e) neural backpropagating, in the logic encoder, a difference between the fixed-size incorrectly encoded tree and the fixed-size correctly encoded tree; and
performing after said increasing the accuracy of the logic encoder:
deploying, into a production environment, the logic encoder without the encoded tree generator;
generating, by the logic encoder, a second fixed-size encoded logic for a second source logic in an integrated development environment (IDE); and
performing, by the IDE based on the second fixed-size encoded logic, source logic completion.
9 . The one or more non-transitory computer-readable media of claim 8 wherein:
said inferring the fixed-size incorrectly encoded tree is performed by the encoded tree generator;
the instructions further cause adjusting the encoded tree generator based on said difference between the fixed-size incorrectly encoded tree and the fixed-size correctly encoded tree.
10 . The one or more non-transitory computer-readable media of claim 8 wherein the instructions further cause sizing, based on a count of distinct tree paths in parse trees in a training corpus, the fixed-size correctly encoded tree and the fixed-size incorrectly encoded tree.
11 . The one or more non-transitory computer-readable media of claim 8 wherein said inferring the fixed-size incorrectly encoded tree comprises inferring a plurality of distinct tree paths in the parse tree that represents the source logic.
12 . The one or more non-transitory computer-readable media of claim 11 wherein said inferring the plurality of distinct tree paths comprises inferring a respective count of each tree path in the plurality of distinct tree paths.
13 . The one or more non-transitory computer-readable media of claim 8 wherein said generating the fixed-size correctly encoded tree is based on at least one selected from a group consisting of: a predefined minimum path length, a predefined maximum path length, and non-terminal nodes of the parse tree that represents the source logic but not terminal nodes.
14 . The one or more non-transitory computer-readable media of claim 8 wherein the instructions further cause based on the second fixed-size encoded logic, performing at least one selected from a group consisting of: source logic translation, source logic documentation generation, and programing reference material recommendation.