Dynamic instrumentation via user-level mechanisms

1. A method of instrumenting a source process that calls target processes, comprising:

accessing a loaded but paused source process executable;

disassembling the source process executable to identify one or more system calls to be instrumented and, for a respective system call of the one or more system calls, an adjacent relocatable instruction,

wherein a first address of the adjacent relocatable instruction is within a distance from a second address of the respective system call that is reachable by a given jump instruction;

instrumenting the respective system call, wherein instrumenting the respective system call comprises:

building a trampoline for the respective system call, the trampoline comprising a check flag instruction and two areas of the trampoline that are selectively executed according to results of the check flag instruction, the two areas of the trampoline comprising:

a first area of the trampoline comprising instructions to execute a relocated copy of the adjacent relocatable instruction and return flow to an address immediately following the adjacent relocatable instruction, and

a second area of the trampoline comprising instructions to invoke a handler in association with executing a relocated copy of the respective system call and return flow to an address immediately following the respective system call;

overwriting the adjacent relocatable instruction with at least a set flag instruction and a jump-to-trampoline instruction; and;

overwriting the respective system call with a jump instruction that is directed to the jump-to-trampoline instruction, thereby forming a jump sequence that produces a different state of a flag when flow jumps from the respective system call than when flow proceeds sequentially to the jump-to-trampoline instruction;

causing execution of the source process executable with the trampoline operable to instrument the one or more system calls.

2. The method of claim 1 , further comprising pausing the source process executable and creating a copy of the source process executable for disassembly.

3. The method of claim 1 , wherein the given jump instruction is of a code length in bytes that is less than or equal to a code length of the respective system call.

4. The method of claim 1 , wherein a code length in bytes of the adjacent relocatable instruction is sufficient to hold at least a set flag instruction and a jump-to-trampoline instruction.

5. The method of claim 1 , further comprising:

the first area of the trampoline comprising a clear flag instruction that clears a flag that is set when the flow proceeds sequentially to the jump-to-trampoline instruction.

6. The method of claim 1 , further comprising:

overwriting the adjacent relocatable instruction with a short jump that bypasses the set flag instruction when the flow proceeds sequentially to the jump-to-trampoline instruction;

invoking, by the jump sequence from the respective system call, the set flag instruction; and

wherein the second area of the trampoline includes a clear flag instruction that clears a flag that is set when the flow proceeds sequentially to the jump-to-trampoline instruction.

7. The method of claim 1 , further comprising:

loading the handler and the trampoline into a region of memory allocated to the source process.

8. The method of claim 1 , wherein the instructions in the second area of the trampoline further include instructions configured to set a context for the handler to instrument the respective system call.

9. The method of claim 1 , wherein the one or more system calls to be instrumented are filtered based on instructions that are received from a user.

10. The method of claim 1 , the second area of the trampoline further comprising instructions that:

invoke the handler with on entry parameters before executing the respective system call and invoke the handler with on exit parameters after executing the respective system call.

11. The method of claim 1 , further comprising:

collecting, from the handler, results associated with instrumenting the one or more system calls;

reporting the results to a user.

12. The method of claim 1 , further comprising:

pausing the source process executable;

removing the instrumentation of the respective system call;

restoring the respective system call and the adjacent relocatable instruction;

resuming execution of the source process executable without the instrumentation.

13. The method of claim 1 , wherein the adjacent relocatable instruction does not include a contained address within code space of the adjacent relocatable instruction that is identified by the disassembling as a target of a jump or a branch from another part of the source process executable.

14. The method of claim 1 , wherein the source process executable is loaded on an Intel x86 processor and the respective system call is a SYSCALL instruction.

15. A system including at least one processor and memory coupled to the processor, the memory loaded with computer program instructions that, when executed on the at least one processor, cause the at least one processor to carry out a method of instrumenting a source process that calls target processes, comprising:

pausing execution of a source process executable;

disassembling the source process executable to identify one or more system calls to be instrumented and, for a respective system call of the one or more system calls, an adjacent relocatable instruction that is within a short jump distance from the respective system call and that occupies sufficient bytes of executable code space to hold at least a set flag instruction and a jump-to-trampoline instruction;

instrumenting the respective system call using an executable object code trampoline to invoke one or more handlers, wherein instrumenting the respective system call includes:

building the executable object code trampoline for the respective system call, the executable object code trampoline comprising a check flag instruction at or near an entry point to the executable object code trampoline and two areas of the executable object code trampoline that are selectively executable depending on results of the check flag instruction, the two areas comprising:

a first area of the executable object code trampoline that is executed responsive to a first state of a flag that includes at least a relocated copy of the adjacent relocatable instruction and a first jump instruction that returns flow to an address immediately following the adjacent relocatable instruction in the source process executable,

a second area of the executable object code trampoline that is executed responsive to a second state of the flag that includes at least instructions that invoke a handler in association with executing a relocated copy of the respective system call, an instruction that executes the relocated copy of the respective system call, and a second jump instruction that returns flow to an address immediately following the respective system call in the source process executable, and

wherein either the first area or the second area includes a flag reset instruction that changes the flag between the first state and the second state;

relocating the respective system call and the adjacent relocatable instruction to the executable object code trampoline, and

overwriting the respective system call and the adjacent relocatable instruction with a sequence of jump instructions, comprising:

overwriting the adjacent relocatable instruction with at least the set flag instruction and the jump-to-trampoline instruction, and

overwriting the respective system call with a short jump instruction directed to the jump-to-trampoline instruction thereby forming a jump sequence that produces a different state of the flag when flow jumps from the respective system call than when flow proceeds sequentially to the jump-to-trampoline instruction;

causing execution of the source process executable with the executable object code trampoline operable to instrument the one or more system calls.

16. A non-transitory computer readable medium, loaded with computer program instructions that, when executed on one or more processors, cause the processors to carry out a method of instrumenting a source process that calls target processes, the method comprising:

accessing a loaded but paused source process executable;

disassembling the source process executable to identify one or more system calls to be instrumented and, for a respective system call of the one or more system calls, an adjacent relocatable instruction,

wherein a first address of the adjacent relocatable instruction is within a distance from a second address of the respective system call that is reachable by a given jump instruction;

instrumenting the respective system call, wherein instrumenting the respective system call includes:

building a trampoline for the respective system call, the trampoline comprising a check flag instruction and two areas of the trampoline that are selectively executed according to results of the check flag instruction, the two areas of the trampoline comprising:

a first area of the trampoline comprising instructions to execute a relocated copy of the adjacent relocatable instruction and return flow to an address immediately following the adjacent relocatable instruction, and

a second area of the trampoline comprising instructions to invoke a handler in association with executing a relocated copy of the respective system call and return flow to an address immediately following the respective system call;

overwriting the adjacent relocatable instruction with at least a set flag instruction and a jump-to-trampoline instruction;

overwriting the respective system call with a jump instruction that is directed to the jump-to-trampoline instruction thereby forming a jump sequence that produces a different state of a flag when flow jumps from the respective system call than when flow proceeds sequentially to the jump-to-trampoline instruction;

causing execution of the source process executable with the trampoline operable to instrument the one or more system calls.

17. The non-transitory computer readable medium of claim 16 , wherein the first area of the trampoline includes a clear flag instruction that clears a flag set when the flow proceeds sequentially to the jump-to-trampoline instruction.

18. The non-transitory computer readable medium of claim 16 , the method further comprising:

overwriting the adjacent relocatable instruction with a short jump that bypasses the set flag instruction when the flow proceeds sequentially to the jump-to-trampoline instruction;

invoking, by the jump sequence from the respective system call, the set flag instruction; and

wherein the second area of the trampoline includes a clear flag instruction that clears a flag that is set when the flow proceeds sequentially to the jump-to-trampoline instruction.

19. The non-transitory computer readable medium of claim 16 , the method further comprising:

loading the handler and the trampoline into a region of memory allocated to the source process.

20. The non-transitory computer readable medium of claim 16 , wherein the instructions in the second area that invoke the handler further include instructions that set a context for the handler to instrument the respective system call.