Compiler caching based on semantic structure
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for how a build system uses import graphs to maintain a current compilation cache and determine when compilation targets should be recompiled. A request is received to compile a compilation target. A plurality of files are identified that are used to build the compilation target. An import graph that represents import dependencies among the files used the build the compilation target is generated for the compilation target. The import graph is traversed to assign a respective identifier to each node in the import graph. A cache key is generated from data representing import relationships represented by the import graph and data representing contents of files used to build the compilation target. If the cache key is invalid or does not exist, compiling the compilation target.
1. A computer-implemented method comprising:
receiving a request to compile a compilation target;
identifying a plurality of files used to build the compilation target;
generating an import graph for the compilation target that represents import dependencies among the files used to build the compilation target, wherein each node of the import graph represents a file and each edge represents that a first file represented by a source node of the edge imports a second file represented by a target node of the edge;
traversing the import graph to assign a respective identifier to each node in the import graph;
generating a cache key from data representing import relationships represented by the import graph and data representing contents of the files used to build the compilation target;
determining, from a compilation cache, whether the cache key is valid; and
compiling the compilation target using the files if and only if the cache key for the compilation target is invalid or does not exist in the compilation cache.
2. The computer-implemented method of claim 1 , wherein data representing contents of the files represented by the nodes in the import graph does not include import statements occurring in the files represented by the nodes in the import graph.
3. The computer-implemented method of claim 1 , wherein the cache key does not depend on file names of the files or any file names represented by nodes in the import graph.
4. The computer-implemented method of claim 1 , wherein the cache key does not depend on a directory structure of files represented by nodes in the import graph.
5. The computer-implemented method of claim 1 , further comprising:
renaming or moving a source code file represented by one or more nodes in the import graph;
recomputing a cache key for the renamed or moved source code file;
comparing the recomputed cache key to a previously-generated cache key for the source code file;
determining that the recomputed cache key is the same as the previously-generated cache key; and
providing a compiled version of the source code file from the compilation cache based on the recomputed cache key without recompiling the source code file.
6. The computer-implemented method of claim 1 , wherein a cache key from data representing import relationships between files represented by the nodes in the import graph and data representing contents of the files represented by the nodes in the import graph includes: (i) an identifier of the node representing the file; (ii) identifiers of any nodes representing source code files that are directly imported by the file; and (iii) a portion of the contents of the source code file.
7. The computer-implemented method of claim 1 , further comprising:
adding a compiled received source code file to a cache of a caching infrastructure using the generated cache key as the cache key to the file.
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving a request to compile a compilation target;
identifying a plurality of files used to build the compilation target;
generating an import graph for the compilation target that represents import dependencies among the files used to build the compilation target, wherein each node of the import graph represents a file and each edge represents that a first file represented by a source node of the edge imports a second file represented by a target node of the edge;
traversing the import graph to assign a respective identifier to each node in the import graph;
generating a cache key from data representing import relationships represented by the import graph and data representing contents of the files used to build the compilation target;
determining, from a compilation cache, whether the cache key is valid; and
compiling the compilation target using the files if and only if the cache key for the compilation target is invalid or does not exist in the compilation cache.
9. The system of claim 8 , wherein data representing contents of the files represented by the nodes in the import graph does not include import statements occurring in the files represented by the nodes in the import graph.
10. The system of claim 8 , wherein the cache key does not depend on file names of the files or any file names represented by nodes in the import graph.
11. The system of claim 8 , wherein the cache key does not depend on a directory structure of the files represented by nodes in the import graph.
12. The system of claim 8 , wherein the operations further comprise:
renaming or moving a source code file represented by one or more nodes in the import graph;
recomputing a cache key for the renamed or moved source code file;
comparing the recomputed cache key to a previously-generated cache key for the source code file;
determining that the recomputed cache key is the same as the previously-generated cache key; and
providing a compiled version of the source code file from the compilation cache based on the recomputed cache key without recompiling the source code file.
13. The system of claim 8 , wherein a cache key from data representing import relationships between files represented by the nodes in the import graph and data representing contents of the files represented by the nodes in the import graph includes: (i) an identifier of the node representing the file; (ii) identifiers of any nodes representing source code files that are directly imported by the file; and (iii) a portion of the contents of the source code file.
14. The system of claim 8 , wherein the operations further comprise:
adding a compiled received source code file to a cache of a caching infrastructure using the generated cache key as the cache key to the file.
15. One or more machine-readable hardware storage devices encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving a request to compile a compilation target;
identifying a plurality of files used to build the compilation target;
generating an import graph from the compilation target that represents import dependencies among the files used to build the compilation target, wherein each node of the import graph represents a file and each edge represents that a first file represented by a source node of the edge imports a second file represented by a target node of the edge;
traversing the import graph to assign a respective identifier to each node in the import graph;
generating a cache key from data representing import relationships represented by the import graph and data representing contents of the files used to build the compilation target;
determining, from a compilation cache, whether the cache key is valid; and
compiling the compilation target using the files if and only if the cache key for the compilation target is invalid or does not exist in the compilation cache.
16. The one or more machine-readable hardware storage devices of claim 15 , wherein data representing contents of the files represented by the nodes in the import graph does not include import statements occurring in the files represented by the nodes in the import graph.
17. The one or more machine-readable hardware storage devices of claim 15 , wherein the cache key does not depend on file names of the files or any file names represented by nodes in the import graph.
18. The one or more machine-readable hardware storage devices of claim 15 , wherein the cache key does not depend on a directory structure of files represented by nodes in the import graph.
19. The one or more machine-readable hardware storage devices of claim 15 , wherein the operations further comprise:
renaming or moving a source code file represented by one or more nodes in the import graph;
recomputing a cache key for the renamed or moved source code file;
comparing the recomputed cache key to a previously-generated cache key for the source code file;
determining that the recomputed cache key is the same as the previously-generated cache key; and
providing a compiled version of the source code file from the compilation cache based on the recomputed cache key without recompiling the source code file.
20. The one or more machine-readable hardware storage devices of claim 15 , wherein a cache key from data representing import relationships between files represented by the nodes in the import graph and data representing contents of the files represented by the nodes in the import graph includes: (i) an identifier of the node representing the file; (ii) identifiers of any nodes representing source code files that are directly imported by the file; and (iii) a portion of the contents of the source code file.