Automatic correctness and performance measurement of binary transformation systems
Disclosed is a test engine intended to evaluate the correctness and measure the performance effects of a binary transformation technique. The disclosed system takes source code as input and compiler information/flags as input. The transformation-under-test is applied to the compiler, creating a transformed compiler. A random test case generator residing within the test engine for injecting illegal code structures to modify the project source code, build flags, or compiler's operating environment, thereby creating an unlimited number of input test cases for the compiler. The test engine compiles the source code utilizing both the raw and transformed compilers and compares the results. For example, the test engine renders a pass/fail judgement on the binary transformation based on a metric of near equivalence between the results of the raw compiler and transformed compiler. By using one or more bitmasks, the evaluation process factors in differences attributed to compiler run-time generated artifacts.
1. A method of assessing the effectiveness and correctness of a binary transformation technique for cyberhardening program binaries against future cyberattacks, comprising:
receiving a plurality of project source files at a source code repository;
generating a transformed compiler by applying at least one binary transformation technique on a raw compiler;
generating raw compiler test results data and transformed compiler test results data based on processing, using one or more computer processors, the plurality of project source files through the raw compiler and the transformed compiler;
applying one or more bitmasks that are configured for selectively ignoring build-specific metadata included in the raw compiler test results data and the transformed compiler test results data;
performing a comparison of the raw compiler test results data and the transformed compiler test results data; and
assessing impact of the at least one binary transformation technique based on characterizing computational performance of the raw compiler and the transformed compiler.
2. The method of claim 1 , wherein the build-specific metadata included in the raw compiler test results data and the transformed compiler test results data includes information related to source project ID, compiler version, or time stamp.
3. The method of claim 1 , wherein the processing the plurality of project source files through the raw compiler and the transformed compiler are further based on injecting, in the plurality of project source files, randomly-generated test cases for simulating one or more of: illegal code structures, illegal compiler flags, and illegal combinations of input files and compiler flags.
4. The method of claim 3 , wherein the illegal code structures include unrecognizable input, corrupted files, wrong reserved words, or non-textual elements.
5. The method of claim 3 , wherein the illegal compiler flags include flags are configured to generate an illegal operation error, in a case when the illegal compiler flags are used by the raw compiler or the transformed compiler during execution.
6. The method of claim 5 , wherein the illegal compiler flags include one or more of: mistyping flags, incompatible flags that do not exist in a dictionary associated with the raw compiler or the transformed compiler, or missing files from flags that require a file input.
7. The method of claim 3 , wherein the illegal combinations of input files and compiler flags include corrupted source files, unrecognizable input, wrong reserved words or structures, or non-textual elements.
8. The method of claim 1 , wherein the at least one binary transformation technique included Basic Block Randomization (BBR), Stack Frame Randomization (SFR), Control-Flow Integrity (CFI), or a combination thereof.
9. The method of claim 1 , wherein the plurality of project source files include source code written in a high-level language, compiler options and flags, compiler identification information, a target operating system running on the one or more computer processors, an architecture of the one or more computer processors, or project metadata.
10. The method of claim 1 , wherein the target operating system running on the one or more computer processors includes Windows, Ubuntu, MacOS, or Linux, and wherein the architecture of the one or more computer processors include Intel x86 or x64, embedded systems with 8, 16, or 32-bit processor cores, or Power PC.
11. The method of claim 1 , wherein the computational performance of the raw compiler and the transformed compiler is characterized based on memory utilization metrics, compilation time metrics, or a metric associated with utilization of the one or more processors.
12. The method of claim 1 , wherein the build-specific metadata includes static build-time artifacts that remain unchanged in repeated compilations of the plurality of project source files regardless of whether the raw compiler or the transformed compiler is used for the compilations.
13. The method of claim 1 , wherein the build-specific metadata includes dynamic build-time artifacts that change over repeated compilations of the plurality of project source files when the raw compiler or the transformed compiler is used for the compilations.
14. The method of claim 1 , wherein the one or more bitmasks is a project-specific bitmask, further comprising:
computing differences between a binary generated from compilation on the raw compiler and a binary generated from compilation on the transformed compiler; and
upon determining from the differences that the binary generated from compilation on the raw compiler and the binary generated from compilation on the transformed compiler are functionally equal, generating the project-specific bitmask particularly applicable to the plurality of project source files.
15. The method of claim 1 , wherein the one or more bitmasks is a compiler bitmask, further comprising:
determining location of artifact in a data file associated used in compilation of the plurality of project source files; and
upon determining the artifact, generating the compiler bitmask applicable to the plurality of project source files and another plurality of project source files.